{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kats 205 Forecasting with Global Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial will introduce how to use the global model in Kats.  The global model is a new and powerful forecasting method that combines exponential smoothing models with recurrent neural networks, resulting in higher accuracy than other approaches. The table of contents for Kats 205 is as follows:\n",
    "\n",
    "1. Overview of Global Model for Forecasting  \n",
    "2. Build Your Own Global Model or Global Ensemble From Scratch  \n",
    "    2.1 Introduction to `GMParam`  \n",
    "    2.2 Forecasting using a single global model with `GMModel`  \n",
    "    2.3 Forecasting using a global model ensemble with `GMEnsemble`  \n",
    "    2.4 Backtesting with `GMBacktester` \n",
    "3. Using Pretrained Global Model or Global Ensemble"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** We provide two types of tutorial notebooks\n",
    "- **Kats 101**, basic data structure and functionalities in Kats \n",
    "- **Kats 20x**, advanced topics, including advanced forecasting techniques, advanced detection algorithms, `TsFeatures`, meta-learning, global model etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Overview of Global Model for Forecasting\n",
    "\n",
    "The Global Model, henceforth abbreviated as GM, is a powerful forecasting model that was originally proposed by Slawek Smyl and won the Computational Intelligence in Forecasting International Time Series Competition (2016) and the M4 Forecasting competition (2018).  The GM effectively combines exponential smoothing models with neural networks in a way that results in higher accuracy than a method that only uses pure statistics or machine learning.  \n",
    "\n",
    "Kats has many forecasting models that only use pure statistics, and we discussed many of these approaches in [Kats 201](kats_201_forecasting.ipynb).  We also provide an ML approach to forecasting in our metalearning framework for forecasting, which is covered in [Kats 204](kats_204_metalearning.ipynb).  The GM approach generates more accurate forecasts by combining the advantages of pure statistical and ML approaches.  For more details on how this works, please see the [original paper for the GM](https://www.sciencedirect.com/science/article/pii/S0169207019301153).\n",
    "\n",
    "GM is trained with a large number of time series of the same time granularity.  It generally supports batch processing, meaning it can efficiently generate forecasts for several time series at the same time.  \n",
    "\n",
    "\n",
    "In Kats, we build upon the original model and allow support for two different neural network types when building a GM:\n",
    "1. **Recurrent neural network (RNN)**: This is the default and it's best for short-term forecasting.  \n",
    "2. **Sequence to Sequence (S2S)**: More optimal for medium to long term forecasting. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Build Your Own Global Model or Global Ensemble From Scratch\n",
    "\n",
    "The `GMModel` is our basic class to build a single GM.  Kats also supports Global Model Ensembles (GMEs), which are ensembles of independent GMs, with the `GMEnsemble` class.  The `GMParam` is the parameter class for both `GMModel` and `GMEnsemble`.  We also provide the `GMBacktester` class for parameter tunning and backtesting. \n",
    "\n",
    "The examples in this section are designed to display the basic functionality of each of the aforementioned classes.  They are of limited scale and not expected to provide good performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Introduction to `GMParam`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All parameters for a GM or a GME are specified using the `GMParam` class.  The `GMParam` class does basic parameter checking when initialized to ensure that the parameters are correctly specified.  Here are some of its key arguments:\n",
    "\n",
    "* **freq**: `str` or `pd.Timedelta`, The time granularity of the model (and the input time series.) For example, `freq='D'` indicates a daily model;\n",
    "* **model_type**: `str`, The name of neural network type - either 'rnn' (recurrent neural network) or 's2s' (sequence to sequence). Default is 'rnn';\n",
    "* **seasonality**: `int`, The integer length of the seasonality period. The default value is 1, indicating a non-seasonal model;\n",
    "* **input_window**: `int`, This parameter specifies the size into which we segment our input time series to feed them into the neural network.  It should be greater than the `seasonality` argument;\n",
    "* **fcst_window**: `int`,  The number of data points forecast in a single forecast step;\n",
    "* **quantile**: `list[float]`, The float values of the quantiles to forecast.  The first value of this list should always be 0.5, representing the median.  The default value is `[0.5,0.05,0.95,0.99]`;\n",
    "* **nn_structure**: `list[list[int]]`, The structure of the neural network. If not specified, the default value is `[[1,3]]`;\n",
    "* **loss_function**: `list[str]`, The name of loss function - either 'pinball' or 'adjustedpinball';\n",
    "* **gmfeature**: `list[str]` or `str`; A single or a list of feature names.\n",
    "\n",
    "For the definition of other parameters, please see our documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "# For Google Colab:\n",
    "!pip install kats\n",
    "!wget https://raw.githubusercontent.com/facebookresearch/Kats/main/kats/data/air_passengers.csv\n",
    "!wget https://globalmodel.s3.amazonaws.com/pretrained_daily_rnn.p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import sys\n",
    "import warnings\n",
    "import os\n",
    "import pprint\n",
    "\n",
    "warnings.simplefilter(action='ignore')\n",
    "sys.path.append(\"../\")\n",
    "\n",
    "from kats.models.globalmodel.utils import GMParam"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we initialize a `GMParam` instance that we will use to train a daily model with weekly seasonality. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "gmparam = GMParam(\n",
    "    input_window = 35, \n",
    "    fcst_window = 31,\n",
    "    seasonality = 7,\n",
    "    freq = 'D',\n",
    "    loss_function = 'adjustedpinball',\n",
    "    nn_structure = [[1,3]],\n",
    "    gmfeature = ['last_date'],\n",
    "    epoch_num = 1, \n",
    "    epoch_size = 2, # use a small num just for demonstration\n",
    "    gmname = \"daily_default\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Forecasting using a single global model with `GMModel`  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from kats.models.globalmodel.model import GMModel, load_gmmodel_from_file\n",
    "from kats.models.globalmodel.serialize import global_model_to_json, load_global_model_from_json\n",
    "from kats.consts import TimeSeriesData"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to train a global model.  A `GMModel` object can be initialized with just one parameter - an instance of the `GMParam` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "gm = GMModel(gmparam)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train a `GMModel` object, we need a list or a dictionary of `TimeSeriesData` objects. We will simulate two dictionaries, one for training and one for testing, using the `get_ts` method from our test functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from kats.tests.models.test_globalmodel import get_ts\n",
    "\n",
    "train_TSs = [get_ts(n*5, '2020-05-06', has_nans=False) for n in range(20, 40)]\n",
    "test_TSs = [get_ts(n*2, '2020-05-06', has_nans=False) for n in range(40, 45)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is straightforward to train the GM using the `train` function.  This function also saves basic information about the training process, which we look at below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'train_loss_monitor': [0.106872484],\n",
      " 'train_loss_val': [0.2538221776485443],\n",
      " 'valid_fcst_monitor': [],\n",
      " 'valid_loss_monitor': [{'epoch': 0}]}\n"
     ]
    }
   ],
   "source": [
    "# train the model\n",
    "training_info = gm.train(train_TSs)\n",
    "\n",
    "#training_info saves the information of training process\n",
    "pprint.pprint(training_info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use the trained model to generate forecasts using the `predict` function.  The input of the `predict` function can either be a single `TimeSeriesData` or a list/dictionary of them.  The `predict` function also requires you to specify the number of steps you wish to forecast.  Here, we demonstrate doing batch forecasting on the 5 `TimeSeriesData` objects in `test_TSs` for 3 steps.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "fcsts = gm.predict(test_TSs, steps = 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This generates a dictionary with 5 keys, one for each time series in `test_TSs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys([0, 1, 2, 3, 4])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fcsts.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The values in the dictionary are the 3-step forecasts for each time series in `test_TSs`.  Let's look at the forecast for the 3rd time series in the list.  We see a `pd.DataFrame` that gives forecasts for each percentiles 5, 50, 95, and 99 in the 3 days after the time series ends."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fcst_quantile_0.5</th>\n",
       "      <th>fcst_quantile_0.05</th>\n",
       "      <th>fcst_quantile_0.95</th>\n",
       "      <th>fcst_quantile_0.99</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.423670</td>\n",
       "      <td>-0.408314</td>\n",
       "      <td>0.243321</td>\n",
       "      <td>0.548310</td>\n",
       "      <td>2020-07-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.776328</td>\n",
       "      <td>1.326708</td>\n",
       "      <td>1.187961</td>\n",
       "      <td>1.242450</td>\n",
       "      <td>2020-08-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.179862</td>\n",
       "      <td>0.634731</td>\n",
       "      <td>-0.205554</td>\n",
       "      <td>0.198845</td>\n",
       "      <td>2020-08-02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fcst_quantile_0.5  fcst_quantile_0.05  fcst_quantile_0.95  \\\n",
       "0           0.423670           -0.408314            0.243321   \n",
       "1           1.776328            1.326708            1.187961   \n",
       "2           0.179862            0.634731           -0.205554   \n",
       "\n",
       "   fcst_quantile_0.99       time  \n",
       "0            0.548310 2020-07-31  \n",
       "1            1.242450 2020-08-01  \n",
       "2            0.198845 2020-08-02  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fcsts[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can save the model using the `save_model` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save model\n",
    "gm.save_model(\"gm_example_1.p\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the saved model and again use it to repeat the forecast we did above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fcst_quantile_0.5</th>\n",
       "      <th>fcst_quantile_0.05</th>\n",
       "      <th>fcst_quantile_0.95</th>\n",
       "      <th>fcst_quantile_0.99</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.423670</td>\n",
       "      <td>-0.408314</td>\n",
       "      <td>0.243321</td>\n",
       "      <td>0.548310</td>\n",
       "      <td>2020-07-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.776328</td>\n",
       "      <td>1.326708</td>\n",
       "      <td>1.187961</td>\n",
       "      <td>1.242450</td>\n",
       "      <td>2020-08-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.179862</td>\n",
       "      <td>0.634731</td>\n",
       "      <td>-0.205554</td>\n",
       "      <td>0.198845</td>\n",
       "      <td>2020-08-02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fcst_quantile_0.5  fcst_quantile_0.05  fcst_quantile_0.95  \\\n",
       "0           0.423670           -0.408314            0.243321   \n",
       "1           1.776328            1.326708            1.187961   \n",
       "2           0.179862            0.634731           -0.205554   \n",
       "\n",
       "   fcst_quantile_0.99       time  \n",
       "0            0.548310 2020-07-31  \n",
       "1            1.242450 2020-08-01  \n",
       "2            0.198845 2020-08-02  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load model\n",
    "gm2 = load_gmmodel_from_file(\"gm_example_1.p\")\n",
    "\n",
    "# make prediction \n",
    "fcsts2 = gm2.predict(test_TSs, steps = 3)\n",
    "fcsts2[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's remove the saved model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.remove(\"gm_example_1.p\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also encode GM into a json string using the  `global_model_to_json` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "gm_str = global_model_to_json(gm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's repeat the same prediction one more time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fcst_quantile_0.5</th>\n",
       "      <th>fcst_quantile_0.05</th>\n",
       "      <th>fcst_quantile_0.95</th>\n",
       "      <th>fcst_quantile_0.99</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.423670</td>\n",
       "      <td>-0.408314</td>\n",
       "      <td>0.243321</td>\n",
       "      <td>0.548310</td>\n",
       "      <td>2020-07-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.776328</td>\n",
       "      <td>1.326708</td>\n",
       "      <td>1.187961</td>\n",
       "      <td>1.242450</td>\n",
       "      <td>2020-08-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.179862</td>\n",
       "      <td>0.634731</td>\n",
       "      <td>-0.205554</td>\n",
       "      <td>0.198845</td>\n",
       "      <td>2020-08-02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fcst_quantile_0.5  fcst_quantile_0.05  fcst_quantile_0.95  \\\n",
       "0           0.423670           -0.408314            0.243321   \n",
       "1           1.776328            1.326708            1.187961   \n",
       "2           0.179862            0.634731           -0.205554   \n",
       "\n",
       "   fcst_quantile_0.99       time  \n",
       "0            0.548310 2020-07-31  \n",
       "1            1.242450 2020-08-01  \n",
       "2            0.198845 2020-08-02  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load model from json string\n",
    "gm3 = load_global_model_from_json(gm_str)\n",
    "\n",
    "# make prediction \n",
    "fcsts3 = gm3.predict(test_TSs, steps = 3)\n",
    "fcsts3[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Forecasting using a global model ensemble with `GMEnsemble`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also easily build an ensemble of several independently trained GMs with `GMEnsemble` class.  We initialize the `GMEnsemble` class with the same `GMParam` object that we use to initialize a single GM, but there are a few other parameters needed to indicate how to create the ensemble.\n",
    "\n",
    "Here is the list of attributes needed to initialize GMEnsemble`:\n",
    "* **gmparam**: `GMParam`; the parameters for each GM in the ensemble;\n",
    "* **ensemble_type**: `str`, how to aggregate the forecasts - either 'median' or 'mean'.  Default is 'median';\n",
    "* **splits**: `int`, the number of sub-datasets into which the training data is to be partitioned.  Default is 3;\n",
    "* **overlap**: `bool`, Whether or not sub-datasets overlap with each other or not. If True, each training examples appears in `splits-1` sub-datasets.  Otherwise, each training example appears in only 1 sub-dataset.  Default is True;\n",
    "* **replicate**: `int`, The number of GMs in the ensemble to be trained on each sub-dataset. Default is 1;\n",
    "* **multi**: `bool`, whether or not to use multi-processing for training and prediction. Default is False.\n",
    "\n",
    "Note that a `GMEnsemble` object will build `splits*replicate` independent `GMModel` objects, and the final forecasts are aggregated from the forecasts generated from each trained `GMModel` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from kats.models.globalmodel.ensemble import GMEnsemble, load_gmensemble_from_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can initialize a `GMEnsemble` object as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "gme = GMEnsemble(gmparam, splits=3, overlap=True, replicate=1, multi=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can train the `GMEnsemble` object. Note that one has the choice of setting aside a test set from the training data to measure the performance of each `GMModel` object throughout the training process; we do so using the `test_size` paramter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[W ParallelNative.cpp:206] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)\n",
      "[W ParallelNative.cpp:206] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)\n",
      "[W ParallelNative.cpp:206] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)\n"
     ]
    }
   ],
   "source": [
    "gme.train(train_TSs, test_size = 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Information about the training process for each GM in the ensemble can be access using the `gm_info` attribute.  This attribute will contain a list of dictionaries; one dictionary for each GM in the ensemble.  Since we set `splits = 3` and `replicate = 1`, we should expect this list to have `splits * replicate = 3*1 = 3` dictionaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(gme.gm_info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at each of those dictionaries now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'test_info': [      smape     sbias  exceed_0.05  exceed_0.95  exceed_0.99  step  idx  epoch\n",
      "0  1.443195 -1.177987     0.322581     0.709677     0.645161     0   15      0\n",
      "1  1.458923 -0.540060     0.483871     0.548387     0.483871     0    5      0],\n",
      " 'train_loss_monitor': [0.10591892],\n",
      " 'train_loss_val': [0.21183784306049347],\n",
      " 'valid_fcst_monitor': [],\n",
      " 'valid_loss_monitor': [{'epoch': 0}]}\n",
      "\n",
      "{'test_info': [      smape     sbias  exceed_0.05  exceed_0.95  exceed_0.99  step  idx  epoch\n",
      "0  1.489374 -1.057571     0.258065     0.741935     0.774194     0   15      0\n",
      "1  1.494057 -0.396561     0.290323     0.548387     0.451613     0    5      0],\n",
      " 'train_loss_monitor': [0.09518992],\n",
      " 'train_loss_val': [0.28556976467370987],\n",
      " 'valid_fcst_monitor': [],\n",
      " 'valid_loss_monitor': [{'epoch': 0}]}\n",
      "\n",
      "{'test_info': [      smape     sbias  exceed_0.05  exceed_0.95  exceed_0.99  step  idx  epoch\n",
      "0  1.289125 -0.724129     0.193548     0.774194     0.806452     0   15      0\n",
      "1  1.391239 -0.197581     0.322581     0.516129     0.612903     0    5      0],\n",
      " 'train_loss_monitor': [0.09116545],\n",
      " 'train_loss_val': [0.3418704718351364],\n",
      " 'valid_fcst_monitor': [],\n",
      " 'valid_loss_monitor': [{'epoch': 0}]}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in range(len(gme.gm_info)):\n",
    "    pprint.pprint(gme.gm_info[i])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that each dictionary has a key called 'test_info', which stores a list of DataFrames showing how each GM in the ensemble performs on each example of the test data.  Each DataFrame in the list represents a single epoch.\n",
    "\n",
    "Since we set `epoch_num = 1` in the parameters, each list will only have one DataFrame.  Since the length of `train_TSs` is 20 and we set `test_size = 0.1`, there are two elements in our test set.  For each item in the test data set, in addition to the symmetric MAPE and bias metrics, the DataFrame shows the percentage of forecasted values that exceed all of the specified quantiles (in the `GMParam` object) other than the median.  Since the quantiles we are looking at other than the median are 0.05, 0.95 and 0.95, we get additional metrics `exceed_0.05`, `exceed_0.95`, and `exceed_0.99`.  \n",
    "\n",
    "Let's take a look at one of these DataFrames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>smape</th>\n",
       "      <th>sbias</th>\n",
       "      <th>exceed_0.05</th>\n",
       "      <th>exceed_0.95</th>\n",
       "      <th>exceed_0.99</th>\n",
       "      <th>step</th>\n",
       "      <th>idx</th>\n",
       "      <th>epoch</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.443195</td>\n",
       "      <td>-1.177987</td>\n",
       "      <td>0.322581</td>\n",
       "      <td>0.709677</td>\n",
       "      <td>0.645161</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.458923</td>\n",
       "      <td>-0.540060</td>\n",
       "      <td>0.483871</td>\n",
       "      <td>0.548387</td>\n",
       "      <td>0.483871</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      smape     sbias  exceed_0.05  exceed_0.95  exceed_0.99  step  idx  epoch\n",
       "0  1.443195 -1.177987     0.322581     0.709677     0.645161     0   15      0\n",
       "1  1.458923 -0.540060     0.483871     0.548387     0.483871     0    5      0"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gme.gm_info[0]['test_info'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After training the `GMEnsemble` object, you now can use it to generate forecasts. Similar to the `GMModel` object, the input of the `predict` function can either be a single `TimeSeriesData` or a list/dictionary of them and you must also specify the number of steps you wish to forecast."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "fcsts=gme.predict(test_TSs, steps = 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As above with `GMModel`, this generates a dictionary with 5 keys, one for each time series in `test_TSs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys([0, 1, 2, 3, 4])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fcsts.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at one of the forecasts.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fcst_quantile_0.5</th>\n",
       "      <th>fcst_quantile_0.05</th>\n",
       "      <th>fcst_quantile_0.95</th>\n",
       "      <th>fcst_quantile_0.99</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.631042</td>\n",
       "      <td>0.762085</td>\n",
       "      <td>1.562355</td>\n",
       "      <td>1.102208</td>\n",
       "      <td>2020-07-29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.302816</td>\n",
       "      <td>-0.978175</td>\n",
       "      <td>-0.852965</td>\n",
       "      <td>-0.105852</td>\n",
       "      <td>2020-07-30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.818922</td>\n",
       "      <td>-0.865128</td>\n",
       "      <td>-0.537662</td>\n",
       "      <td>-0.214839</td>\n",
       "      <td>2020-07-31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fcst_quantile_0.5  fcst_quantile_0.05  fcst_quantile_0.95  \\\n",
       "0           0.631042            0.762085            1.562355   \n",
       "1          -0.302816           -0.978175           -0.852965   \n",
       "2          -0.818922           -0.865128           -0.537662   \n",
       "\n",
       "   fcst_quantile_0.99       time  \n",
       "0            1.102208 2020-07-29  \n",
       "1           -0.105852 2020-07-30  \n",
       "2           -0.214839 2020-07-31  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fcsts[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can save a `GMEnsemble` to a file similarly to how we did for `GMModel`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fcst_quantile_0.5</th>\n",
       "      <th>fcst_quantile_0.05</th>\n",
       "      <th>fcst_quantile_0.95</th>\n",
       "      <th>fcst_quantile_0.99</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.631042</td>\n",
       "      <td>0.762085</td>\n",
       "      <td>1.562355</td>\n",
       "      <td>1.102208</td>\n",
       "      <td>2020-07-29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.302816</td>\n",
       "      <td>-0.978175</td>\n",
       "      <td>-0.852965</td>\n",
       "      <td>-0.105852</td>\n",
       "      <td>2020-07-30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.818922</td>\n",
       "      <td>-0.865128</td>\n",
       "      <td>-0.537662</td>\n",
       "      <td>-0.214839</td>\n",
       "      <td>2020-07-31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fcst_quantile_0.5  fcst_quantile_0.05  fcst_quantile_0.95  \\\n",
       "0           0.631042            0.762085            1.562355   \n",
       "1          -0.302816           -0.978175           -0.852965   \n",
       "2          -0.818922           -0.865128           -0.537662   \n",
       "\n",
       "   fcst_quantile_0.99       time  \n",
       "0            1.102208 2020-07-29  \n",
       "1           -0.105852 2020-07-30  \n",
       "2           -0.214839 2020-07-31  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# save model\n",
    "gme.save_model(\"gme_example_1.p\")\n",
    "\n",
    "# load model\n",
    "gme2 = load_gmensemble_from_file(\"gme_example_1.p\")\n",
    "\n",
    "# remove the saved model\n",
    "os.remove(\"gme_example_1.p\")\n",
    "\n",
    "# generate forecasts\n",
    "fcsts2=gme.predict(test_TSs, steps = 3)\n",
    "fcsts2[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When encoding a `GMEnsemble` into a JSON string, we need to clear the `gm_info` attribute because it contains dictionaries with `pd.DataFrame` objects, which are not serializable.  Otherwise the code is the same as for `GMModel`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fcst_quantile_0.5</th>\n",
       "      <th>fcst_quantile_0.05</th>\n",
       "      <th>fcst_quantile_0.95</th>\n",
       "      <th>fcst_quantile_0.99</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.631042</td>\n",
       "      <td>0.762085</td>\n",
       "      <td>1.562355</td>\n",
       "      <td>1.102208</td>\n",
       "      <td>2020-07-29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.302816</td>\n",
       "      <td>-0.978175</td>\n",
       "      <td>-0.852965</td>\n",
       "      <td>-0.105852</td>\n",
       "      <td>2020-07-30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.818922</td>\n",
       "      <td>-0.865128</td>\n",
       "      <td>-0.537662</td>\n",
       "      <td>-0.214839</td>\n",
       "      <td>2020-07-31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fcst_quantile_0.5  fcst_quantile_0.05  fcst_quantile_0.95  \\\n",
       "0           0.631042            0.762085            1.562355   \n",
       "1          -0.302816           -0.978175           -0.852965   \n",
       "2          -0.818922           -0.865128           -0.537662   \n",
       "\n",
       "   fcst_quantile_0.99       time  \n",
       "0            1.102208 2020-07-29  \n",
       "1           -0.105852 2020-07-30  \n",
       "2           -0.214839 2020-07-31  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# encode model into json string\n",
    "\n",
    "gme.gm_info=None # Need to remove this because pd.DataFrame is not serilizable\n",
    "gme_str = global_model_to_json(gme)\n",
    "\n",
    "# load model from json string\n",
    "gme3 = load_global_model_from_json(gme_str)\n",
    "\n",
    "# generate forecasts\n",
    "fcsts3=gme.predict(test_TSs, steps = 3)\n",
    "fcsts3[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Backtesting with `GMBacktester`  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `GMBacktester` object helps evaluate the hyperparameter setting (i.e., the `GMParam` object). Here is a list of some of the attributes:\n",
    "* **data**: `list[TimeSeriesData` or `dict[TimeSeriesData]`, A list or a dictionary of time series objects for training and validation;\n",
    "* **gmparam**: `GMParam`, the parameters that we're testing;\n",
    "* **backtest_timestamp**: `list[str]` or `list[pd.Timestamp]`, timestamps used to split the time series for training and validation;\n",
    "* **splits**: `int`, the number of sub-datasets into which the training data is to be partitioned.  Default is 3;\n",
    "* **overlap**: `bool`,  Whether or not sub-datasets overlap with each other or not. If True, each training examples appears in `splits-1` sub-datasets.  Otherwise, each training example appears in only 1 sub-dataset.  Default is True;\n",
    "* **replicate**: `int`, The number of GMs in the ensemble to be trained on each sub-dataset.  Default is 1;\n",
    "* **test_size**: `float`, The proportion of the input `data` to be used for testing.  Default is 0.1.\n",
    "\n",
    "\n",
    "For the full list of attributes, please see our documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from kats.models.globalmodel.backtester import GMBackTester"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can initialize the `GMBackTester` object as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "gbm = GMBackTester(train_TSs, gmparam, backtest_timestamp = ['2020-08-10'], test_size=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now one can run backtesting using the `run_backtest` function.  This will give you a DataFrame that shows the evaluation metrics for each GM that was generated for each time series in our test data set.  For each backtesting date, number of GMs created is equal to `splits*replicate`, and the backtester also calculates the evaluation metrics for an ensemble that takes the median of each of these GMs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>smape</th>\n",
       "      <th>sbias</th>\n",
       "      <th>exceed_0.05</th>\n",
       "      <th>exceed_0.95</th>\n",
       "      <th>exceed_0.99</th>\n",
       "      <th>model_num</th>\n",
       "      <th>step</th>\n",
       "      <th>idx</th>\n",
       "      <th>type</th>\n",
       "      <th>backtest_ts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.569974</td>\n",
       "      <td>-0.916378</td>\n",
       "      <td>0.258065</td>\n",
       "      <td>0.516129</td>\n",
       "      <td>0.451613</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>single</td>\n",
       "      <td>2020-08-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.540258</td>\n",
       "      <td>-0.601911</td>\n",
       "      <td>0.225806</td>\n",
       "      <td>0.419355</td>\n",
       "      <td>0.419355</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>single</td>\n",
       "      <td>2020-08-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.440604</td>\n",
       "      <td>-0.266532</td>\n",
       "      <td>0.290323</td>\n",
       "      <td>0.516129</td>\n",
       "      <td>0.483871</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>single</td>\n",
       "      <td>2020-08-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.471687</td>\n",
       "      <td>-0.670066</td>\n",
       "      <td>0.258065</td>\n",
       "      <td>0.516129</td>\n",
       "      <td>0.451613</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>ensemble</td>\n",
       "      <td>2020-08-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.618392</td>\n",
       "      <td>-0.772541</td>\n",
       "      <td>0.354839</td>\n",
       "      <td>0.612903</td>\n",
       "      <td>0.580645</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>single</td>\n",
       "      <td>2020-08-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.486467</td>\n",
       "      <td>-0.549642</td>\n",
       "      <td>0.290323</td>\n",
       "      <td>0.580645</td>\n",
       "      <td>0.548387</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>single</td>\n",
       "      <td>2020-08-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1.496364</td>\n",
       "      <td>-0.212388</td>\n",
       "      <td>0.387097</td>\n",
       "      <td>0.645161</td>\n",
       "      <td>0.580645</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>single</td>\n",
       "      <td>2020-08-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1.487712</td>\n",
       "      <td>-0.621116</td>\n",
       "      <td>0.354839</td>\n",
       "      <td>0.612903</td>\n",
       "      <td>0.580645</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>ensemble</td>\n",
       "      <td>2020-08-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1.423745</td>\n",
       "      <td>-0.250940</td>\n",
       "      <td>0.612903</td>\n",
       "      <td>0.032258</td>\n",
       "      <td>0.032258</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>single</td>\n",
       "      <td>2020-08-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1.575215</td>\n",
       "      <td>-0.331045</td>\n",
       "      <td>0.612903</td>\n",
       "      <td>0.032258</td>\n",
       "      <td>0.032258</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>single</td>\n",
       "      <td>2020-08-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1.613945</td>\n",
       "      <td>0.134239</td>\n",
       "      <td>0.677419</td>\n",
       "      <td>0.096774</td>\n",
       "      <td>0.096774</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>single</td>\n",
       "      <td>2020-08-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1.511595</td>\n",
       "      <td>-0.186997</td>\n",
       "      <td>0.612903</td>\n",
       "      <td>0.032258</td>\n",
       "      <td>0.032258</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>ensemble</td>\n",
       "      <td>2020-08-10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       smape     sbias  exceed_0.05  exceed_0.95  exceed_0.99  model_num  \\\n",
       "0   1.569974 -0.916378     0.258065     0.516129     0.451613        0.0   \n",
       "1   1.540258 -0.601911     0.225806     0.419355     0.419355        1.0   \n",
       "2   1.440604 -0.266532     0.290323     0.516129     0.483871        2.0   \n",
       "3   1.471687 -0.670066     0.258065     0.516129     0.451613        NaN   \n",
       "4   1.618392 -0.772541     0.354839     0.612903     0.580645        0.0   \n",
       "5   1.486467 -0.549642     0.290323     0.580645     0.548387        1.0   \n",
       "6   1.496364 -0.212388     0.387097     0.645161     0.580645        2.0   \n",
       "7   1.487712 -0.621116     0.354839     0.612903     0.580645        NaN   \n",
       "8   1.423745 -0.250940     0.612903     0.032258     0.032258        0.0   \n",
       "9   1.575215 -0.331045     0.612903     0.032258     0.032258        1.0   \n",
       "10  1.613945  0.134239     0.677419     0.096774     0.096774        2.0   \n",
       "11  1.511595 -0.186997     0.612903     0.032258     0.032258        NaN   \n",
       "\n",
       "    step  idx      type backtest_ts  \n",
       "0      0    4    single  2020-08-10  \n",
       "1      0    4    single  2020-08-10  \n",
       "2      0    4    single  2020-08-10  \n",
       "3      0    4  ensemble  2020-08-10  \n",
       "4      0   10    single  2020-08-10  \n",
       "5      0   10    single  2020-08-10  \n",
       "6      0   10    single  2020-08-10  \n",
       "7      0   10  ensemble  2020-08-10  \n",
       "8      1   10    single  2020-08-10  \n",
       "9      1   10    single  2020-08-10  \n",
       "10     1   10    single  2020-08-10  \n",
       "11     1   10  ensemble  2020-08-10  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "backtest_df = gbm.run_backtest()\n",
    "backtest_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the backtesting data for a single one of the GMs that we just trained.  We can do so as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>smape</th>\n",
       "      <th>sbias</th>\n",
       "      <th>exceed_0.05</th>\n",
       "      <th>exceed_0.95</th>\n",
       "      <th>exceed_0.99</th>\n",
       "      <th>model_num</th>\n",
       "      <th>step</th>\n",
       "      <th>idx</th>\n",
       "      <th>type</th>\n",
       "      <th>backtest_ts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.540258</td>\n",
       "      <td>-0.601911</td>\n",
       "      <td>0.225806</td>\n",
       "      <td>0.419355</td>\n",
       "      <td>0.419355</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>single</td>\n",
       "      <td>2020-08-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.486467</td>\n",
       "      <td>-0.549642</td>\n",
       "      <td>0.290323</td>\n",
       "      <td>0.580645</td>\n",
       "      <td>0.548387</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>single</td>\n",
       "      <td>2020-08-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1.575215</td>\n",
       "      <td>-0.331045</td>\n",
       "      <td>0.612903</td>\n",
       "      <td>0.032258</td>\n",
       "      <td>0.032258</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>single</td>\n",
       "      <td>2020-08-10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      smape     sbias  exceed_0.05  exceed_0.95  exceed_0.99  model_num  step  \\\n",
       "1  1.540258 -0.601911     0.225806     0.419355     0.419355        1.0     0   \n",
       "5  1.486467 -0.549642     0.290323     0.580645     0.548387        1.0     0   \n",
       "9  1.575215 -0.331045     0.612903     0.032258     0.032258        1.0     1   \n",
       "\n",
       "   idx    type backtest_ts  \n",
       "1    4  single  2020-08-10  \n",
       "5   10  single  2020-08-10  \n",
       "9   10  single  2020-08-10  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "backtest_df[backtest_df.model_num == 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the test time series have different lengths, we may need a different number of prediction steps for each time series.  The table we see above shows a different row for each training step.  In addition to the length of each time series, the number of training steps is determined by the `fcst_window` parameter in the `GMParam` object.  If we want to get a single metric for each time series, we can do the following aggregation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>smape</th>\n",
       "      <th>sbias</th>\n",
       "      <th>exceed_0.05</th>\n",
       "      <th>exceed_0.95</th>\n",
       "      <th>exceed_0.99</th>\n",
       "      <th>model_num</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>idx</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.540258</td>\n",
       "      <td>-0.601911</td>\n",
       "      <td>0.225806</td>\n",
       "      <td>0.419355</td>\n",
       "      <td>0.419355</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1.530841</td>\n",
       "      <td>-0.440344</td>\n",
       "      <td>0.451613</td>\n",
       "      <td>0.306452</td>\n",
       "      <td>0.290323</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        smape     sbias  exceed_0.05  exceed_0.95  exceed_0.99  model_num\n",
       "idx                                                                      \n",
       "4    1.540258 -0.601911     0.225806     0.419355     0.419355        1.0\n",
       "10   1.530841 -0.440344     0.451613     0.306452     0.290323        1.0"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "backtest_df[backtest_df.model_num == 1].drop('step', axis=1).groupby('idx').mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the metrics calculated for only the GM Ensemble.  The results are similar to the ones we saw above for a single GM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>smape</th>\n",
       "      <th>sbias</th>\n",
       "      <th>exceed_0.05</th>\n",
       "      <th>exceed_0.95</th>\n",
       "      <th>exceed_0.99</th>\n",
       "      <th>model_num</th>\n",
       "      <th>step</th>\n",
       "      <th>idx</th>\n",
       "      <th>type</th>\n",
       "      <th>backtest_ts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.471687</td>\n",
       "      <td>-0.670066</td>\n",
       "      <td>0.258065</td>\n",
       "      <td>0.516129</td>\n",
       "      <td>0.451613</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>ensemble</td>\n",
       "      <td>2020-08-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1.487712</td>\n",
       "      <td>-0.621116</td>\n",
       "      <td>0.354839</td>\n",
       "      <td>0.612903</td>\n",
       "      <td>0.580645</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>ensemble</td>\n",
       "      <td>2020-08-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1.511595</td>\n",
       "      <td>-0.186997</td>\n",
       "      <td>0.612903</td>\n",
       "      <td>0.032258</td>\n",
       "      <td>0.032258</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>ensemble</td>\n",
       "      <td>2020-08-10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       smape     sbias  exceed_0.05  exceed_0.95  exceed_0.99  model_num  \\\n",
       "3   1.471687 -0.670066     0.258065     0.516129     0.451613        NaN   \n",
       "7   1.487712 -0.621116     0.354839     0.612903     0.580645        NaN   \n",
       "11  1.511595 -0.186997     0.612903     0.032258     0.032258        NaN   \n",
       "\n",
       "    step  idx      type backtest_ts  \n",
       "3      0    4  ensemble  2020-08-10  \n",
       "7      0   10  ensemble  2020-08-10  \n",
       "11     1   10  ensemble  2020-08-10  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "backtest_df[backtest_df.type == 'ensemble']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can do the same aggregation here if we want to get a single metric for each time series using the ensemble forecasts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>smape</th>\n",
       "      <th>sbias</th>\n",
       "      <th>exceed_0.05</th>\n",
       "      <th>exceed_0.95</th>\n",
       "      <th>exceed_0.99</th>\n",
       "      <th>model_num</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>idx</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.471687</td>\n",
       "      <td>-0.670066</td>\n",
       "      <td>0.258065</td>\n",
       "      <td>0.516129</td>\n",
       "      <td>0.451613</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1.499653</td>\n",
       "      <td>-0.404057</td>\n",
       "      <td>0.483871</td>\n",
       "      <td>0.322581</td>\n",
       "      <td>0.306452</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        smape     sbias  exceed_0.05  exceed_0.95  exceed_0.99  model_num\n",
       "idx                                                                      \n",
       "4    1.471687 -0.670066     0.258065     0.516129     0.451613        NaN\n",
       "10   1.499653 -0.404057     0.483871     0.322581     0.306452        NaN"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "backtest_df[backtest_df.type == 'ensemble'].drop('step', axis=1).groupby('idx').mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are some constraints on how we pick the backtesting days.  Specifically, we need to have enough training days based on the the settings that we specify in the `GMParam` object.  We can see how many days that is using the `min_train_length` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "90"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gbm.min_train_length"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on our current parameters, this tells us we need at least 90 data points for training.  If we try a date sooner than 90 days out from the first date, we will get an error.  The first date in each of our training time series is '2020-05-06', so the following does not work because '2020-08-01' is less than 90 days out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# THIS DOES NOT WORK\n",
    "\n",
    "# gbm = GMBackTester(train_TSs, gmparam, backtest_timestamp = ['2020-08-01'], test_size=0.1)\n",
    "\n",
    "# gbm.run_backtest()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Using Pretrained Global Model or Global Ensemble"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Kats, we provide two pre-trained daily `GMEnsemble` objects.  One ensemble is a S2S-GME and and the other one is a RNN-GME, and you can find them [here](https://globalmodel.s3.amazonaws.com/pretrained_daily_s2s.p) and [here](https://globalmodel.s3.amazonaws.com/pretrained_daily_rnn.p) respecitvely. Both of them are trained with M4 dataset. One can use them for forecasting exploration or benchmark.\n",
    "\n",
    "\n",
    "We can load these ensembles the same way we loaded the ensembles we created in section 2.3 using the `load_gmensemble_from_file function`.  We will show an example of how to do this for the RNN-GME (but the process would be the same for RNN-GME).  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Begin by downloading the `pretrained_daily_rnn.p` file by clicking [this link](https://globalmodel.s3.amazonaws.com/pretrained_daily_rnn.p) and moving the file to any directory.  For simplicity, we will move the file to the `Tutorials` directory of Kats for this exercise.  Then we can load it using the `load_gmensemble_from_file` function as we did above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<kats.models.globalmodel.ensemble.GMEnsemble at 0x7ff4b0f6afa0>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gme_rnn = load_gmensemble_from_file(\"pretrained_daily_rnn.p\")\n",
    "gme_rnn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: You need to download the `pretrained_daily_rnn.p` file give the complete or relative path to the file as an argument to the `load_gmensemble_from_file` function.  We give the relative path `\"pretrained_daily_rnn.p\"` above because we put the file in the `Tutorials` directory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have loaded the ensemble, we can use it to make predictions the same way we did with our own forecasts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fcst_quantile_0.5</th>\n",
       "      <th>fcst_quantile_0.01</th>\n",
       "      <th>fcst_quantile_0.05</th>\n",
       "      <th>fcst_quantile_0.95</th>\n",
       "      <th>fcst_quantile_0.99</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.681012</td>\n",
       "      <td>0.089251</td>\n",
       "      <td>0.321020</td>\n",
       "      <td>0.987761</td>\n",
       "      <td>1.220350</td>\n",
       "      <td>2020-07-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.012009</td>\n",
       "      <td>0.211353</td>\n",
       "      <td>0.512495</td>\n",
       "      <td>1.475238</td>\n",
       "      <td>1.762766</td>\n",
       "      <td>2020-08-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.323513</td>\n",
       "      <td>-0.498166</td>\n",
       "      <td>-0.204912</td>\n",
       "      <td>0.818574</td>\n",
       "      <td>1.102585</td>\n",
       "      <td>2020-08-02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fcst_quantile_0.5  fcst_quantile_0.01  fcst_quantile_0.05  \\\n",
       "0           0.681012            0.089251            0.321020   \n",
       "1           1.012009            0.211353            0.512495   \n",
       "2           0.323513           -0.498166           -0.204912   \n",
       "\n",
       "   fcst_quantile_0.95  fcst_quantile_0.99       time  \n",
       "0            0.987761            1.220350 2020-07-31  \n",
       "1            1.475238            1.762766 2020-08-01  \n",
       "2            0.818574            1.102585 2020-08-02  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fcsts = gme_rnn.predict(test_TSs, steps = 3)\n",
    "fcsts[3]"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "5b6e8fba36db23bc4d54e0302cd75fdd75c29d9edcbab68d6cfc74e7e4b30305"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
